v0.9.74: | KAOS

Compare Source

What's Changed

🎉 Exciting New Features

☁️ Google cloud storage source

You can now effortlessly configure your data integration source blocks to Google Cloud Storage using this snippet. It specifies the destination bucket, sets the file type to Parquet, and provides the path to your service account credentials needed for authentication.

by @​TalaatHasanin in https://github.com/mage-ai/mage-ai/pull/5334

🌬️ 🍽️ Airtable Integration

Effortlessly streamline your data data integration source blocks with Airtable using this tailored script. It establishes a connection to your Airtable base, designates the specific table for integration, and securely integrates your Airtable API key for access.

by @​TalaatHasanin in https://github.com/mage-ai/mage-ai/pull/5404

✂️ Add trim reformat action to transformer block

A new "trim reformat" action has been added to the Python transformer block, allowing for the removal of leading and trailing whitespace from specified text columns. This enhancement ensures cleaner and more consistent data formatting by automatically stripping unnecessary spaces around text entries.

by @​cristopheridlc in https://github.com/mage-ai/mage-ai/pull/5321

🐻‍❄️ Enable polars dataframe in GCS data exporter

This update enhances the export method of the GCS IO module, expanding its functionality to support exporting Polars DataFrames in addition to the previously supported Pandas DataFrames. This new capability allows users to seamlessly work with both data formats, offering greater flexibility in managing and exporting data from their workflows.

by @​TalaatHasanin in https://github.com/mage-ai/mage-ai/pull/5348

Add support to scheduler name on k8s executor

This update adds the ability to customize Kubernetes scheduler options when using the Kubernetes Executor. Users can now directly configure scheduler settings, providing increased flexibility and control over how pods are scheduled within Kubernetes environments.

by @​messerzen in https://github.com/mage-ai/mage-ai/pull/5412

🐛 Bug Fixes

• [jk] Update condition for cookie secure property by @​johnson-mage in https://github.com/mage-ai/mage-ai/pull/5327
• [xy] Fix NoneType in trigger settings. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5329
• [xy] Fix dbt block delete request with nginx. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5332
• [xy] Fix API for file versions. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5338
• [xy] Fix schema creation when schema name is wrapped with double quotes. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5366
• [jk] Update test ENV value by @​johnson-mage in https://github.com/mage-ai/mage-ai/pull/5372
• [xh] Add retry logic and throw if open AI API returns error by @​matrixstone in https://github.com/mage-ai/mage-ai/pull/5375
• [jk] Fix typo for custom frequency in trigger review by @​johnson-mage in https://github.com/mage-ai/mage-ai/pull/5392
• [td] Fix Git syncs API error returning sync_on_executor_start by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5388
• #​5417 Fix bug: Can't use git with Bitbucket by @​kennynguyeenx in https://github.com/mage-ai/mage-ai/pull/5418
• Dynamic block fixes
  • [td] Fix index used in dynamic block by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5373
  • [xy] Fix the condition of checking dynamic upstreams completed mage-ai/mage-ai@2f2eeca
  • [xy] Some updates and fixes for dynamic blocks mage-ai/mage-ai@7eba140
• [hw] Fix a missing import issue with data types in a template for Data Exporters by @​csharplus in https://github.com/mage-ai/mage-ai/pull/5377
• SASL_SSL kafka Host Name Validation by @​sujiplr in https://github.com/mage-ai/mage-ai/pull/5380

💅 Enhancements & Polish

• [xy] Reduce CPU usage in scheduler. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5331
• [xy] Pass main pod labels to workspaces. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5344
• enhancement: bump duckdb to 1.0.0 by @​matsonj in https://github.com/mage-ai/mage-ai/pull/5358
• Pass all SSL settings through for SASL_SSL by @​csharplus in https://github.com/mage-ai/mage-ai/pull/5381
• Update unit tests to use the methods compatible with Python 3.12 by @​csharplus in https://github.com/mage-ai/mage-ai/pull/5398
• Introduce StrEnum and IntEnum classes to resolve unit test failures with Python 3.11+ by @​csharplus in https://github.com/mage-ai/mage-ai/pull/5397
• Add NATS credentials file authentication by @​ddecaro11 in https://github.com/mage-ai/mage-ai/pull/5410
• Update a few unit tests that failed on Python 3.12 by @​csharplus in https://github.com/mage-ai/mage-ai/pull/5420
• Fix kafka module related unit test failure issues in Python 3.12 by @​csharplus in https://github.com/mage-ai/mage-ai/pull/5424
• Add trigger name in kwargs mage-ai/mage-ai@81e4f91
• Expose column name related config to postgres sink config mage-ai/mage-ai@c40af5f
• [xh] Add ENV variables to turn on/off AI API by @​matrixstone in https://github.com/mage-ai/mage-ai/pull/5394

Other Changes

• Update amazon-s3.mdx by @​Ayumi08 in https://github.com/mage-ai/mage-ai/pull/5337
• Fix spelling in compute-resource.mdx by @​samacciu23 in https://github.com/mage-ai/mage-ai/pull/5356
• Update logger.py by @​vffv2000 in https://github.com/mage-ai/mage-ai/pull/5383
• [hw] Update the user guide for Kafka streaming with latest changes based on testing by @​csharplus in https://github.com/mage-ai/mage-ai/pull/5378
• Update the development document to resolve one issue encountered by @​csharplus in https://github.com/mage-ai/mage-ai/pull/5396
• Fixing spelling error by @​benjaminhawn in https://github.com/mage-ai/mage-ai/pull/5407

New Contributors

• @​cristopheridlc made their first contribution in https://github.com/mage-ai/mage-ai/pull/5321
• @​TalaatHasanin made their first contribution in https://github.com/mage-ai/mage-ai/pull/5348
• @​samacciu23 made their first contribution in https://github.com/mage-ai/mage-ai/pull/5356
• @​matsonj made their first contribution in https://github.com/mage-ai/mage-ai/pull/5358
• @​vffv2000 made their first contribution in https://github.com/mage-ai/mage-ai/pull/5383
• @​benjaminhawn made their first contribution in https://github.com/mage-ai/mage-ai/pull/5407
• @​ddecaro11 made their first contribution in https://github.com/mage-ai/mage-ai/pull/5410
• @​Jaykold made their first contribution in https://github.com/mage-ai/mage-ai/pull/5340
• @​kennynguyeenx made their first contribution in https://github.com/mage-ai/mage-ai/pull/5418
• @​messerzen made their first contribution in https://github.com/mage-ai/mage-ai/pull/5412

Full Changelog: mage-ai/mage-ai@0.9.73...0.9.74

v0.9.73: | Deadpool & Wolverine 🥷🐺

Compare Source

What's Changed

🎉 Exciting New Features

🧠 Memory management upgrade

This feature enhances system performance and stability by optimizing allocation, deallocation, and overall utilization of RAM resources, resulting in improved efficiency and reduced memory-related errors.

Check out the doc for details.

🔄 Dynamic blocks 2.0

Introducing an enhanced Dynamic blocks feature for creating dynamic content blocks that can adapt based on user input or data changes, offering improved flexibility and interactivity for users. This update aims to streamline workflows and enhance the overall user experience by making content more responsive and customizable.

Learn more in this doc.

🔐 Azure DB connection via Key Vault

Securely retrieve a database connection URL stored in Azure Key Vault using environment variables for authentication.

AZURE_KEY_VAULT_URL
AZURE_CLIENT_ID
AZURE_CLIENT_SECRET
AZURE_TENANT_ID

by @​wangxiaoyou1993 in #​5302

👁️ Workspace monitoring

Add an "Overview" page and "Pipeline runs" page to the Workspace Management UI. This provides some overall monitoring for all of the pipeline runs in the different workspaces without having to individually open up each workspace instance.

by @​johnson-mage in https://github.com/mage-ai/mage-ai/pull/5311

🐛 Bug Fixes

• [td] Fix editor v2 page by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5222
• [jk] Add specific lodash dependency version by @​johnson-mage in https://github.com/mage-ai/mage-ai/pull/5237
• [xy] Don't force cast type to int or float. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5254
• [xy] Some fixes for using requests base path. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5265
• [xy] Update the file content API implementation. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5267
• [jk] Default command center item by @​johnson-mage in https://github.com/mage-ai/mage-ai/pull/5275
• [jk] Include base path when loading favicon by @​johnson-mage in https://github.com/mage-ai/mage-ai/pull/5286
• [xy] Only run memory usage failure check when executor type is local python by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5300
• [xy] Fix kernel interruption. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5301
• [jk] Differentiate current_user local storage by @​johnson-mage in https://github.com/mage-ai/mage-ai/pull/5294
• [td] Fix blocks waiting and when fetching upstream w/ no output by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5307

💅 Enhancements & Polish

• [td] Canvas nodes by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5199
• [td] Menus and more by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5227
• Models and API for execution framework by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5230
• [xy] Pipeline execution for execution framework by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5239
• [xy] Interpolate the sql profile. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5252
• [xy] Allow optional block groups. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5279
• [jk] Cookie session updates by @​johnson-mage in https://github.com/mage-ai/mage-ai/pull/5289
• [xy] Add all upstream dbt blocks when adding a dbt block by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5290
• [xy] Support customizing node selector in k8s config. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5304

✨ Other Changes

• [tc] Update releases.mdx by @​thomaschung408 in https://github.com/mage-ai/mage-ai/pull/5234
• Update using-terraform.mdx by @​tk26 in https://github.com/mage-ai/mage-ai/pull/5235
• [xy] Resolve CVE-2024-35225 by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5236
• Add create_dataset parameter to Bigquery #​5197 by @​gracia10 in https://github.com/mage-ai/mage-ai/pull/5218
• [jk] Add back yarn.lock file by @​johnson-mage in https://github.com/mage-ai/mage-ai/pull/5256
• [jk] Remove unused api/cleaner dir by @​johnson-mage in https://github.com/mage-ai/mage-ai/pull/5291
• [td] Remove fake executables by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5308
• [xy] Add export frontend job in github action by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5313
• [xy] Fix export frontend job by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5314
• [xy] Remove circleci job. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5315
• [xy] Create PR instead of push directly to master branch for frontend build. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5317
• [CI] Update frontend build by @​github-actions in https://github.com/mage-ai/mage-ai/pull/5318
• [xy] Bump version to 0.9.73 by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5312

🆕 New Contributors

• @​tk26 made their first contribution in https://github.com/mage-ai/mage-ai/pull/5235
• @​gracia10 made their first contribution in https://github.com/mage-ai/mage-ai/pull/5218
• @​github-actions made their first contribution in https://github.com/mage-ai/mage-ai/pull/5318

Full Changelog: mage-ai/mage-ai@0.9.72...0.9.73

v0.9.72: | House Targaryen 🐉

Compare Source

What's Changed

🎉 Exciting New Features

🎨 Canvas v2

Navigate large or complex layouts with a new canvas that allows developers to zoom, pan, drag, and drop visual features.

by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5193

🐍 Support custom python sink in Streaming pipeline

Support for custom Python sinks in streaming pipelines allows users to write their own streaming sink implementations, providing greater flexibility in how processed data is handled at the end of the pipeline.

by @​lgarcia11 in https://github.com/mage-ai/mage-ai/pull/5174

Allow configuring custom owner credentials

This feature allows users to configure custom owner credentials for their application by setting specific environment variables:

DEFAULT_OWNER_EMAIL
DEFAULT_OWNER_PASSWORD
DEFAULT_OWNER_USERNAME

by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5165

🐛 Bug Fixes

• [xy] Fix delete/update workspace. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5146
• [td] Fix Dockerfile to install faster fifo from source by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5152
• [xy] Fix import and add user to verify_project. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5151
• [xy] Fix auto_clean_name param. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5161
• [xy] Bump the version of dbt-sqlserver to 1.7.4 by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5162
• [xy] Fix api trigger token comparison. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5167
• [td] Update Snowflake timeout to network_timeout by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5173
• [xy] Fix executor type when not specified. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5175
• [td] Disable playright, never finishes no windows server 3.10 by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5190
• [xy] Fix global data product in single project. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5205
• [td] Revert Next _app back before breaking change by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5207
• [td] Put back Monaco Editor files by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5208

💅 Enhancements & Polish

• [xy] Update pypi github action by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5145
• [td] Magic Kernel using server-sent events to execute code from client by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5131
• [td] Standalone Monaco editor by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5160
• [td] Integrate LSP by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5163
• [xy] Allow configuring AZURE_HOST for authentication. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5166
• [xy] Upgrade polars package version by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5176
• [td] Grid layout v2 by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5168
• [td] System browser by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5177
• [xy] Speed up Pipeline get and list. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5187
• [xy] Speed up autocomplete endpoint with cache. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5188
• [td] Browser items API by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5189
• [td] Fix styles and integrate real files by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5184
• [td] Fix app manager and app loading by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5191
• [td] IDE CRUD operations by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5192
• [xy] Custom kubernetes job prefix by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5195
• [xy] Try using xlarge in circleci. by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5206

✨ Other Changes

• Added repo_path to block kwargs and os.environ, improves multi-project feature by @​hugabora in https://github.com/mage-ai/mage-ai/pull/5144
• [td] DLS v2 by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5147
• Updated Variable Name for Cohesion by @​Ayumi08 in https://github.com/mage-ai/mage-ai/pull/5148
• Remove leading whitespace from Telegram notification message (Issue #​4877) by @​Ayumi08 in https://github.com/mage-ai/mage-ai/pull/5153
• docs: fix markdown formatting by @​ValentinOtt in https://github.com/mage-ai/mage-ai/pull/5059
• [tc] Update community-examples.mdx by @​thomaschung408 in https://github.com/mage-ai/mage-ai/pull/5172
• [td] System browser context menu by @​tommydangerous in https://github.com/mage-ai/mage-ai/pull/5178
• [cf] Updates to schedules doc by @​colefreeman in https://github.com/mage-ai/mage-ai/pull/5212
• Revert "[xy] Upgrade polars package version" by @​wangxiaoyou1993 in https://github.com/mage-ai/mage-ai/pull/5213

🆕 New Contributors

• @​Ayumi08 made their first contribution in https://github.com/mage-ai/mage-ai/pull/5148
• @​ValentinOtt made their first contribution in https://github.com/mage-ai/mage-ai/pull/5059
• @​colefreeman made their first contribution in https://github.com/mage-ai/mage-ai/pull/5171
• @​lgarcia11 made their first contribution in https://github.com/mage-ai/mage-ai/pull/5174

Full Changelog: mage-ai/mage-ai@0.9.71...0.9.72